Online, Opt-in Surveys: Fast and Cheap, but are they Accurate?

ABSTRACT It is increasingly common for government and industry organizations to conduct online, opt-in surveys, in part because they are typically fast, inexpensive, and convenient. Online polls, however, a�ract a non-representative set of respondents, and so it is unclear whether results from such surveys generalize to the broader population. �ese non-representative surveys stand in contrast to probability-based sampling methods, such as random-digit dialing (RDD) of phones, which are a staple of traditional survey research. Here we investigate the accuracy of non-representative data by administering an online, fully opt-in poll of social and political a�itudes. Our survey consisted of 49 multiple-choice attitudinal questions drawn from the probability-based, in-person 2012 General Social Survey (GSS) and select RDD phone surveys by the Pew Research Center. To correct for the inherent biases of non-representative data, we statistically adjust estimates via model-based poststrati�cation, a classic statistical tool but one that is only infrequently used for bias correction. Our online survey took less than one-twentieth the time andmoney of traditional RDD polling, and less than one-hundredth the time and money of GSS polling. A�er statistical correction, we �nd the median absolute di�erence between the non-probability-based online survey and the probability-based GSS and Pew studies is 7 percentage points. �is di�erence is considerably larger than if the surveys were all perfect simple random samples drawn from the same population; the gap, however, is comparable to that between the GSS and Pew estimates themselves. Our results suggest that with proper statistical adjustment, online, non-representative surveys are a valuable tool for practitioners in varied domains.